References to background documents refer to items on the “List of References” below.
Externally powered prosthetic hands are typically controlled using electromyographic (EMG) signals. These signals originate from the polarization and depolarization of the muscle membrane during voluntary contractions and can be measured at the skin surface using either dry or wet-type electrodes. The EMG control signal can be derived from a single site [References 1, 2, 3] or from multiple sites. Past studies have employed two [Reference 4, 5, 6], three [References 7, 8, 9], four [References 5, 10, 11, 12, 13, 14, 15], and up to eight [Reference 16] recording sites with varying levels of success. Englehart and Hudgins argued that four channels of EMG signals are clearly preferable to two [Reference 5]. However, some studies have shown that there is both a practical and theoretical limit to increasing the number of channels [References 10, 5].
MMG Versus EMG
Generally speaking, EMG is a measure of the electrical activity of skeletal muscle. It arises from the successive polarization and depolarization of the muscle membrane. Surface EMG is a temporal and spatial summation of elemental motor unit action potentials.
Muscles vibrate at low frequencies upon contraction. The superficial measurement of these vibrations is known as the mechanomyogram (MMG). The MMG signal originates from the summation of propagated active muscle fibre twitches [Reference 28].
While EMG measures electrical activity, MMG reflects the mechanical activity of the muscle. Hence, EMG and MMG are measuring different but complementary phenomena that accompany muscle contraction.
Static EMG Classifiers
A much sought after goal in EMG-driven prostheses is to provide the user with multiple limb functions, such as hand opening and closing, and wrist rotation. To control multiple functions, it is necessary to map EMG signals corresponding to different muscle contractions to a variety of prosthetic functions. This mapping is commonly achieved by way of a signal classification scheme. In the past decade, many different EMG classification schemes have been proposed for prosthesis control. These classifiers can be divided into four major categories.                Linear classifiers, such as linear discriminant functions, assume that the different classes of contractions are linearly separable [References 10, 5, 17, 2]. Although the linearity assumption would appear limiting, the cited studies have reported competitive performance results, perhaps partly due to the prudent choice of signal features.        Artificial neural network (ANN) classifiers, such as the standard workhorse multilayer feed forward network, can model general nonlinear boundaries among contraction classes [References 1, 4, 6, 7, 3, 8, 9, 11, 13, 14]. Due to this nonlinear capability and robustness to noisy data, these classifiers have been the preferred choice in the vast majority of recent studies. Typically, however, a sufficiently large data set is required to adequately train such networks.        Fuzzy and neuro-fuzzy classifiers deploy combinations of fuzzy coding schemes and nonlinear class modeling of neural networks [References 1, 18, 19, 25]. While not as prevalent as the previous classifier genre, these methods also admit potential nonlinear class boundaries.        K-nearest neighbour EMG classifiers build on the classical nonparametric algorithm by the same name, which is known to be asymptotically Bayes optimal [References 6, 8]. As the computational requirement per classification of a k-nearest classifier is large relative to the aforementioned methods, there have been very few real-time considerations of this genre of classifiers.        
The number of recording sites in past EMG classification systems has mostly varied from 1 to 4. Classification accuracies ranging from 80% to 100% have been reported using ANN classifiers for 4 to 8 different functions. Sebelius et. al. [Reference 26] in a recent study found that modified self-organizing feature map (SOFM) composed of a combination of a Kohonen network and the conscience mechanism algorithm (KNC) was superior in performance to the reference networks, such as multi-layer perceptrons, as regards training time, low memory consumption, and simplicity in finding optimal training parameters and architecture. By applying this method on EMG signals, recorded from 8 sites on an amputee and an able-bodied subject, they could classify six movements out of seven with 100% accuracy. It was found that a KNC network comprising 25 nodes was sufficient to classify the hand movements. The network training was completed after 500 repetitions.
Huang et. al. [Reference 27] used a Gaussian Mixture Model (GMM) based classifier to classify 6 predefined hand movements, using 4 EMG channels. The signals were recorded from 12 able-bodied subjects. The data used in this work were the same as those used in previous investigation by the authors [Reference 11]. Autoregressive coefficients+root-mean-square feature vector was found to be the best feature vector in this application. The performance of GMM was compared to three commonly used classifiers: a linear discriminant analysis, a linear perceptron network, and a multilayer perceptron neural network. The GMM based classifier demonstrated exceptional classification accuracy and resulted in a robust method of motion classification with low computational load. The performance of the system on limb deficient subjects is currently being investigated.
It has been suggested that without the capability to dynamically update over time, the discrimination rate of such majority of aforementioned classifiers would drop to about 60% over the course of actual usage [Reference 4]. It is generally not obvious how previously reported classifiers may be adapted dynamically to accommodate daily variations in EMG signals, due for example, to fatigue.
Dynamic EMG Classifiers
Although most of these methods have only been tested off-line, a few real-time algorithms have been proposed. Englehart et al. [Reference 10] used a pattern recognition approach to process four channels of myoelectric signal, with the task of discriminating four classes of limb movement, based on the “continuous classifier”, introduced in their earlier studies [Reference 5]. Twelve subjects with intact upper limbs participated in the study. The subjects were instructed to perform wrist flexion, wrist extension, radial deviation and ulnar deviation with moderate force. A linear discriminant analysis (LDA) classifier was used in this study. One half of the data was used to train the LDA classifier, and the other half was used as a test set to evaluate the classifiers accuracy.
In a subsequent work, Englehart et. al. [Reference 11] processed four channels of myoelectric signal, with the task of discriminating six classes of limb movement. The training session was strictly divided into the trial consisting of transient data, and the trial consisting of continuous contractions. The main drawback however, of using the transient myoelectric signal as a control input was that it required initiating a contraction from rest. This prohibited switching from class to class in an effective or intuitive manner. It severely impeded the coordination of complex tasks involving multiple degrees of freedom. Therefore, pattern recognition of continuous contraction (steady state signal) was also considered. Eleven subjects with intact limbs participated in the study. The subjects were asked to perform six predetermined limb motions namely, wrist flexion, wrist extension, supination, pronation, palm open, and palm close. Subjects performed two trials as part of this experiment: the first trial consisting of transient data, and the second trial consisting of continuous contractions. In the first trial, subjects were asked to perform rapid contractions, producing 40 burst patterns for each of the six classes, with each burst pattern being 256 samples (256 ms) in length. This was then repeated again to create a second set of patterns. The first set of 6×40=240 patterns was used as a training set, and the second set of 240 patterns used as the test set. In the second trial, subjects underwent four, 60 seconds sessions, producing continuous contractions. Within each session, subjects held each limb motion twice, for a five second duration. The first and third session were used as training sessions. During training sessions, the six limb motions were performed twice, in the order: wrist flexion, wrist extension, supination, pronation, palm open, and palm close. These were performed in response to a computer-generated prompt. These remaining two sessions were used as test sets. In these sessions, each limb motion was also repeated twice but the order of the limb motions was randomized. The transient and continuous data were subject to classification using five different feature sets:                1. the time-domain features [Reference 24];        2. short-time Fourier transform (STFT) coefficients, using a hamming window, 32 ms segments, overlapping by 50%;        3. wavelet transform (WT) coefficients, using a Coiflet-5 mother wavelet;        4. wavelet packet transform (WPT) coefficients, using a Symmlet-5 wavelet and a local discriminant basis algorithm to determine the basis; and        5. Stationary wavelet transform (SWT) coefficients.        
They obtained an impressive 93.25% accuracy using a time-domain feature set, with a multilayer perceptron (MLP) artificial neural network classifier. Although the study only enlisted able-bodied individuals who could perceive the requested predefined movements, future tests with individuals with upper limb deficiencies were to be investigated.
In a separate study, Nishikawa et al. [Reference 4] proposed a novel on-line learning method for discriminating six different predetermined hand movements, namely supination, pronation, flexion, extension, grasping, and opening. They used two electrode channels, which were placed on the inside and the outside of the wrist. However, it is not clear how these locations on the wrist represented the agonist and antagonist muscle groups on the forearm, which are responsible for hand movements. They used a method that had structural similarities to the adaptive heuristic critic (AHC). The AHC generates an inner evaluation signal from external reinforcement signals and input, after which the AHC learns from the inner evaluation signal. This system had three main units:                Analysis unit: This was responsible for extracting the features. In this experiment Gabor transform, which is a discrete Fourier transform using a Gaussian window, was used to generate the feature vector. The interpolation process was used to smooth out the shape of the spectrum.        Adaptation unit: A feed-forward artificial neural network (ANN) was used to realize the Adaptation Unit. This section received the feature vectors from the user, and tried to map them to the desired output.        Trainer unit: The Trainer Unit sent the training data to the Adaptation Unit in order, and updated the inner state of this unit. In order to avoid over learning, Root-mean-square of the teaching signals were calculated and were compared to a predetermined threshold. This served as a prompt for the adaptation unit to either start or stop learning from the training section.        
Although they could discriminate six different forearm movements with 89.9% accuracy, training the prosthetic hand required predetermined movements (subjective judgment). When a subject judged that the system could control a motion intentionally, he taught the hand a new motion, thus directing system learning. On the other hand, when the subject judged that the system was unable to control a motion properly, he had to retrain the motion in question and the system needed to relearn all the target motions. Therefore, the training of the prosthetic hand and adaptive capability were not automatic.
Ajiboye and Weir [Reference 25] used a heuristic fuzzy logic approach to multiple EMG pattern recognition for multifunctional prosthesis control. Heuristics is a computational method that uses trial and error methods to approximate a solution for computationally difficult problems. The task was to classify four different forearm movements in able-bodied subjects, such as wrist extension, wrist flexion, ulnar deviation and finger flexion. The location of the electrode sites were carefully chosen by a trained prosthetist according to the standard clinical practice (four sites for able-bodied, three sites for amputees), and these sites were prepared by shaving any excessive body hair in the site region to maximize physiological separability between signals. The ratio between the number of EMG channels and number of expected movements was considered one. The subject with trauma-induced limb deficiency (Subject A), was not able to perform supination, and the subject with congenital limb deficiency (Subject C) was not able to perform pronation properly. Therefore, based on the ability of the amputee subjects, only three independent surface sites were chosen to classify three different forearm movements. Eight trials were performed by each subject. All contractions were performed at a speed and strength determined by the subject. The actual classification was done offline, by choosing the odd numbered trials as training sets. The real-time experienced was performed on an able-bodied subject, who was believed to be the best-case scenario, as he possessed a full independent control over the motions of interest and was able to obtain adequate physiological separation of EMG signals. A 45.7 millisecond window was chosen to calculate the features. They used mean and standard deviation of the signal for membership function construction, and fuzzy c-means (FCMs) data clustering was used to automate the construction of a simple amplitude-driven inference rule base. With a little bit of tweaking, overall classification rates ranged from 94% to 99%. The fuzzy algorithm also demonstrated success in real-time classification, during both steady state motions and motion state transitioning. Unfortunately, the algorithm was not tested on subjects with limb deficiency in real-time. It also appears that the ability of the subjects to produce self-selected movements was ignored. For example, as discussed earlier, the subjects with limb deficiency were estimated to perform three predefined movements based on their ability to produce forearm movement according to the generally accepted social norms (extension, flexion, and supination (subject C) or pronation (subject A). However, these subjects may be able to produce more movements, which are perceivable to them.
There are several notable patents relevant to the field of the present invention.
U.S. Pat. No. 4,209,860 to Graupe teaches time series modeling, so called “Autoregressive Coefficients” modeling, to model the specific EMG signal produced by a specific movement. As for classification, the method compares the current produced EMG signal with the saved model. It is not clear how the system is trained. However, it was mentioned the calibration was done clinically. This presumably would involve significant user training, and therefore Graupe does not provide a method and prosthetic device wherein the training and adaptive capability is automatic.
U.S. Pat. No. 4,314,379 to Tanie discloses using amplified, rectified, filtered EMG signals from four sites. However, it is mentioned that the number of inputs and outputs can be fixed freely. In accordance with that invention, the user was asked to perform certain movements several times. The results are then averaged these results and these features are fixed in the memory. The upcoming movements would be classified using Linear Discriminant Analysis (LDA), based on the training sets. In this approach the user is restricted to perform certain movements over time, i.e. as part of supervised learning. As for training, it is not clear how many times the user had to train the system and how successfully they could classify movements based on the training data. Apparently, the system could not adaptively change its state over time.
Further, U.S. Pat. No. 6,254,536 to DeVito is an example of systems which are controlled by physiological signals, mainly EEG (brainwaves) and EMG (for monitoring eye movements). The systems include three electrodes in contact with the subject's forehead to pick up the EEG signals. Due to the location of the electrodes on the forehead, brainwave activity (EEG), eye movement and muscle movement (EMG) are all monitored through the same electrodes. It is well known that the frequency band of EEG signals correlates with the state of the brain activity.
For example, alpha wave forms which are usually seen in 8-12 Hz band, show that the subject is relaxed, and high beta wave forms (higher than 18 Hz) show a subject's intensive thinking, such as performing mathematical calculations. In this study, Fast Fourier Transform (FFT) was used to calculate different properties of the signals, and to estimate the state of the brain activity. Passive and active interaction with various electronic media such as video games, movies, music, virtual reality, and computer animations was also disclosed.
U.S. Pat. No. 6,785,574 to Kajitani et al. discusses six predefined actions (forearm pronation, forearm supination, wrist flexion, wrist extension, hand closing and hand opening) to be classified using four sets of electrodes. The subjects were instructed to initiate a 3 second continuous contraction period, followed by a 2 second relaxation period twenty times for each of the six actions. They extracted features while the muscle contraction was maintained (steady state EMG). They set the point when the sum of all EMG signal recorded from four sites, exceeded a pre-determined threshold, which marked the initiation of the contraction. This point served as a time stamp for transient EMG initiation. They waited one second, followed by another 100 milliseconds, to start averaging the steady state EMG signal for each channel. This period of averaging lasted for 1 second. Then, ten averaged values for one muscle contraction were calculated by average the EMG signal over the same period, each shifted and overlapped by 100 milliseconds. These rectified, amplified, filtered signals were subject to logarithmic transformation to get a better resolution, for each channel. However, in practice, depending on how a muscle contracted, there were cases in which the total value of EMG signals did not exceed the preset threshold value. Therefore, not all the twenty training contractions per action could be used as training patterns. On the other hand, setting a threshold is quite subjective. Setting this threshold too low makes the system actuation susceptible to noise, and setting it too high, makes the system not responsive to the EMG actuation. Therefore, the system may not respond to the initiated EMG signals properly and as a result, controlling the powered prostheses, which work based on these algorithms, may become frustrating for the user.
In U.S. Pat. No. 6,859,663 to Kajitani et al., the inventors build on the method taught by U.S. Pat. No. 6,785,574 (referenced directly above) by using redundant coding to encode feature patterns into bit patterns. With respect to arbitrary consecutive values the redundant code always differs by just one bit, differs by two bits when the difference is 2, and differs by three bits when the difference is 3, making it possible to realize a classification circuit with a simple circuit. A circuit that performs high-speed execution of a search technique called a genetic algorithm was used to adaptively rewrite the circuit configuration. Although it was argued that this technique on averaged improved the classification rate by 3.1%, and the required time for circuit synthesis was reduced by 69.6%, the actual accuracy results were not reported. It is also unclear how this method may be beneficial for subjects with limb deficiency in real-time applications. This patent simply provides another example of EMG classification, based on the inventors earlier work in U.S. Pat. No. 6,785,574.
In U.S. Pat. No. 6,272,479 to Farry et al., the inventors argued that the technical problem common to all the reviewed background art in this area was that no program or system provided a reliable and repeatable output when presented with an input signal or signals having features that were not well characterized or linearly separable or separable on a single feature. They solved this problem utilizing an “evolver” program to examine a large number of potential features, which may be from multiple signals to create a “classifier” program. They investigated the following features:                1. mean absolute value (MAV);        2. MAV slope between time intervals;        3. energy;        4. rate of change of energy;        5. average value;        6. number of zero crossings;        7. number of up slopes;        8. number of down slopes;        9. waveform complexity;        10. mean frequency;        11. median frequency;        12. energy or power at any frequency;        13. variance; and        14. autoregressive or autoregressive moving average parameters; and ratios or other combinations of the above.        
The output of the classifier program was compared to the desired output. One or more classifier programs was then created and optimized by the evolver program by means of genetic programming. The desired output was again compared to actual classifier program output and the difference was used as a measure of fitness to guide the evolution of the classifier program. The algorithm required identifying the start command (either manually by the user or automatically by a preset threshold). Then the system tried finding the best possible features and recording sites by means of genetic programming. In real-time implementation, it was suggested that the user should inform the algorithm of any errors by pressing a push button. If the number of errors exceeded a preset value, the algorithm had to retrain itself. There is no indication of the success rate of the proposed algorithm in a real-time application for either limb deficient or able-bodied subjects. Further, it should be noted that this method may or may not converge to the best solution. Genetic programming requires thousands of iterations to search for the both best feature sets and the best classifier function. This process is not usually very fast and requires substantial memory and processing resources.
In addition to the shortcomings mentioned above, there are several other general drawbacks to the previous approaches, which are evident to a person skilled in the art.
In particular, most of the past research and development efforts have been carried out only with able-bodied subjects and not amputees. However, we know that there are significant anatomical and physiological differences between the amputated and intact limb. For example, in congenital amputees, there may be partially developed, missing or fused bones. The soft tissue is commonly heavier or thicker than in the intact limb. In the traumatic amputee, the limb may exhibit scarring, grafted skin areas, muscle atrophy and limited range of motion. As a result, the number of available muscle sites and the quality of useful EMG signals may be very different from those of the intact limb, implying that potentially different EMG features and classifiers may be required.
Further, current devices require users to perform activities according to the generally accepted social norms for executing the target activity. For example, flexion of the forearm muscles is typically associated to prosthetic hand closing. However, these normative modes of activity execution may not be natural for the user and often do not make best use of the individual's abilities. Consequently, effective device usage requires significant user training. It also appears that in most previous approaches, the ability of the subject to express functional intent was also ignored. For instance, the subject would be asked to perform predefined different hand movements, i.e. hand close, hand open, finger movement, etc. which are meaningful for able-bodied subjects but meaningless for amputees, especially those that have congenital limb deficiencies.
In addition, the prior art discloses that pre-processing of EMG signal was necessary. For example, in order to define the start of a specific movement, by examining an EMG signal, a threshold was predefined. For instance, if power of an EMG signal exceeded the threshold, the system would take this point as the time when the movement was initiated. Unfortunately, setting a threshold is quite subjective. Setting this threshold too low makes the system actuation susceptible to noise, and setting it too high, makes the system unresponsive to the EMG actuation. It may be possible to define this threshold by trial and error in a lab, but it is not possible to set this dynamically, as the skin condition (skin perspiration) and other physiological properties of the muscle changes over time, for example, due to fatigue. Therefore, the system may not respond to the initiated EMG signals properly, and as a result, the use of these devices, for example, powered prostheses, becomes frustrating for the user.
Finally, in almost all of the previous approaches, different movements were classified using supervised classifiers (i.e. Artificial Neural Networks). Hence, correct partitioning of the feature space required accurately labelled movements. Such movements could only be obtained according to a strict experimental protocol. In other words, users could not freely generate movement of choice, but had to adhere to the experiment's instruction. Past research often employed pre-recorded EMG databases. The data sets for learning were part of the data sets for verification. These data were recorded with specific electrodes, skin conditions, electrode locations, soft tissue composition, and muscle anatomy. Good performance of the supervised classifiers could only be guaranteed under similar experimental conditions. In other words, to verify the discrimination ability under conditions closer to practical use, the data for verification must be measured and generated after measurement of the data for learning. It has been reported that [Reference 4] if the learning system did not alter its inner state dynamically, the discrimination rate dropped to about 60% in the course of time. On the other hand, ANN classifiers require a great number of samples to train the network. This implied the higher number of training samples. There is also a problem of overtraining the network, which may affect the accuracy of the network. Similarly, genetic programming, used in U.S. Pat. Nos. 6,272,479 and 6,859,663, requires the algorithm to run a great number of times to evaluate thousands of evolutions, which at the end may not converge to the desirable answer. The success of these algorithms greatly depends on the choice of the features and functions. Therefore, these algorithms may not be suitable to be implemented into an embedded system. It is also not clear how fast these algorithms may adaptively change their inner state in embedded real-time applications, where there are limited resources for memory and processing power. Unfortunately, there is no report on the successful implementation of these algorithms on a portable (embedded) system.
What is needed is a method, system and apparatus for classifying muscle signals that overcomes the shortcomings of the prior art.